System and method for generating filters for k-mismatch search

ABSTRACT

A computer system for executing k-mismatch searches, comprising at least one processor, configured to: receive a size of a search term; receive a similarity threshold: and generate a set of filters by: forming the set of filters based on a set of combinations of positions of matches, and gaps for the—size of the search term according to the similarity threshold, wherein each one of the combinations contains at least one filter from the set of filters.

FIELD OF THE DISCLOSED TECHNIQUE

The present invention relates to approximate string matching(k-mismatch) in general, and to generating sets of filters to optimizeavailable computer resources.

BACKGROUND OF THE DISCLOSED TECHNIQUE

Provisional application 62,473,400, titled “Systems and Methods forGenerating a Set of Filters for K-Mismatch Searching”, filed Mar. 19,2017, is herein incorporated by reference in its entirety.

Searching strings over a large-scale database is a time-consuming task.A “naïve algorithm” of a direct search may take days, weeks, or more.Biological databases, such as genome databases or protein databases,collect a large number of genome sequences or proteins, and annotate andanalyze them. Other large-scale databases store pictures, audiorecordings, video, 3D-structures, and the like.

Detecting similarity in nucleotide and amino acid sequences helps toannotate a sequence of interest. For example, if the sequence is a gene,similar sequences may reveal a similarity in function, in regulation, inthe interaction with other molecules, in the three-dimensional structureof the protein product, or in a common evolutionary origin.

A technique to find occurrence of a pattern string in a given text thatmatches approximately is known as the “k-mismatch problem”. Thetechnique is also known as “approximate pattern matching”, “approximatestring matching”, or “fuzzy string searching”. There are severalalgorithms known in the art which solve the k-mismatch problem. Thesealgorithms perform a two stage process in which a search is conductedafter pre-processing of one or more patterns.

Landau and Vishkin disclose a method to solve the K-mismatch problem,also called Kangaroo method, with time complexity of O(n+m) for thepre-processing phase and O(nk) for searching a pattern in the text. G.M. Landau, U. Vishkin, “Efficient string matching in the presence oferrors”, Proc. 26th IEEE FOCS, pp. 126-136, 1985.

Fischer and Paterson disclose an algorithm for the k-mismatch problemwith a time complexity of O(min|Σ|, m)n log m) in “String matching andother products,” Proceedings of the 7th SIAM-AMS Complexity ofComputation, R. Karp, editor, pp. 113-125, 1974.

Abrahamson and Kosaraju disclose a method for the K-mismatch problemthat uses a runtime of O(n √{square root over (m log m)}) in“Generalized string matching”, SIAM journal on Computing,16(6):1039-1051, 1987.

Amir, Lewenstein and Porat disclose another algorithm with runtime ofO(n √{square root over (k log k)} in “Faster algorithms for stringmatching with k mismatches”, Journal of Algorithms, Volume 50, Issue 2,pp 257-275, February 2004. Both Landau and Vishkin's and Amir,Lewenstein and Porat's methods are linear, for small k.

A filtering approach may speed up the runtime of the approximate stringmatching process. Filtering-type algorithms discard large parts of thetext, aiming to retain the significant parts which contain potentialmatches. One common filtering approach is based on q-grams. These aresubstrings of length q. The q-grams of a sequence are all itsoverlapping substrings of length q. Most of the studies in this fieldconcentrate in non-gapped (i.e., usual) q-gram filters.

Applying filtering approach may be optimally carried out by taking intoconsideration a necessary, but not sufficient condition for the wordsimilarity at the selected threshold. For example, being a mammal is anecessary condition for being identified as a human but it is not asufficient condition.

Applying a filtering approach offers improvements over the naïvealgorithm. The following two examples are improvements over the naïvealgorithm for the task of a search for all words in a given text whichare similar to a given word W of 20 letters with one mismatch only (i.e.k=1).

In the first example, a necessary (but not sufficient) similaritycondition is imposed as follows: When dividing the word W into twoparts: W1 (positions 1-10) and W2 (positions 11-20), a mismatch may beeither in W1 (with W2 matched) or in W2 (with W1 matched). A necessarycondition for similarity in this case is an exact match with either W1or with W2. For a search with one-mismatch threshold, one can find wordswhich match exactly with W1 or match exactly with W2 in the text, Thenfor each exact match to W1, the ten subsequent characters to the exactmatch are compared with W2. Similarly, for exact match with W2, the tenprevious characters are compared with W1 (permitting one mismatch). Theexact search of W1 and W2 can be done at time complexity of O(1) bypre-processing the text. This maps all the ten character words,requiring memory of size complexity O(n), with n being the size of thetext. The probability of occurrence of W1 and W2 in the text is (1/d)¹⁰,where d is the size of the alphabet. The time of the search is(n/d)¹⁰×p, where p is the expected amount of random comparisons havingtwo mismatches.

The second example has lower runtime, but it requires much larger sizeof memory, Random Access Memory (“RAM”), for the pre-processing stage.After mapping of all 19-letter words with one “wildcard” in all possiblepositions. Tithe size of this map is about 19×n, and the speed offinding a solution by extracting from W of all 19-letter words with onewildcard is O(1). A disadvantage of such approach is that it isinapplicable for lower thresholds of similarity. For example, forthreshold of 8 mismatches and 20-letter words, the number of possiblewords containing this many wildcards is very high (20:8). This demandssignificant memory resources to store the pre-processing text (more than10⁵×n). This is not feasible for larger value of n.

U.S. Pat. No. 7,756,847 discloses an iterative approach for searchingfor a number of output strings, each having a partial or an exact matchwith a query string in a database comprised of many long strings or along string.

U.S. Pat. No. 7,444,326 discloses approximate substring indexing bydecomposing each string in a database into overlapping “positionalq-grams”, sequences of a predetermined length q, and containinginformation regarding the “position” of each q-gram within the string(i.e., 1st q-gram, 4th q-gram, etc.).

U.S. Pat. No. 7,584,173 discloses a process which determines for asearch string which, if any, of the strings in a text list, has an editdistance from the search string that is less than a threshold. Theprocess uses dynamic programming on a grid with search string characterscorresponding to rows and text characters corresponding to columns.

A generalization of the q-gram filter uses gapped q-grams (also known as“spaced q-grams”), subsets of characters of fixed noncontiguous shapes.The following reference describes how this method is applied.

Randomly selected shapes are disclosed in Califano A., Rigoutsos I.,FLASH: A fast look-up algorithm for string homology, in: Proc. 1stInternational Conference on Intelligent Systems for Molecular Biology(TSMB), pp. 56-64 (1993). Whereas the use of regular shapes aredisclosed in Pevzner, P. A., Waterman, M. S., Multiple Filtration andApproximate Pattern Matching, Algorithmica, 13(1/2), 135-154 (1995); andin Lehtinen, O., Sutinen, E., Tarhio, J., Experiments on Block Indexing,Proceedings of the 3rd South American Workshop on String Processing(WSP'96), Carleton University Press (1996).

Advantages of gapped q-grams on the non-gapped are considered and theirapplication to lossy filtration are disclosed in: Ma, B., Tromp, J., Li,M.: PatternHunter: Faster and More Sensitive Homology Search,Bioinformatics, 18, 440-445 (2002); and Buhler, J., Keich, U., Sun, Y.:Designing Seeds for Similarity Search in Genomic DNA, Proceedings of the7th Annual International Conference on Computational Molecular Biology(RECOMB-03) (April 2003).

Some approaches for gram development are proposed for the case of asmall number of errors in: Farach-Colton M., Landau G., Sahinalp S.,Tsur D., Optimal spaced seeds for faster approximate string matching, J.Comput. System Sci. 73 (7) 1035-1044 (2007); François Nicolas, EricRivals, Hardness of optimal spaced seed design, J. Comput. System Sci.74 (5) 831-849 (2008); and Egidi L., Manzini G.: “Better spaced seedsusing quadratic residues”, Journal of Computer and System Sciences79(7), pp. 1144-1155 (2013).

SUMMARY OF THE DISCLOSED TECHNIQUE

It is an object of the disclosed technique to provide a novel method forgenerating an optimal set of filters for a given task of search, basedon combinatorial properties of shapes, which takes into account theavailable amount of computer resources.

In accordance with the disclosed technique, there is thus provided acomputer system for executing k-mismatch searches, comprising at leastone processor, configured to: receive a size of a search term; receive asimilarity threshold; and generate a set of filters by: forming the setof filters based on a set of combinations of positions of matches andgaps for the size of the search term according to the similaritythreshold, wherein each one of the combinations contains at least onefilter from the set of filters.

In some embodiments, the processor is further configured to create aminimal set of filters by: determining, for each filter of the set offilters, that the filter is a retainable filter when there exists atleast one combination of the set of combinations which contains only thefilter from the set of filters, and deleting the filter from the set offilters when the filter is not a retainable filter.

In some embodiments, each combination of the set of combinationsincludes a match at the first position of the respective combination.

In some embodiments, each combination of the set of combinationsincludes a match at the last position of the respective combination.

In some embodiments, the processor is configured to form the set offilters by: initializing the set of filters; for each selectedcombination of the set of combinations: searching for an occurrence of afilter of the set of filters in the selected combination, deriving a newfilter from the selected combination when the occurrence of a filter ofthe set of filters is not found in the selected combination, and addingthe new filter to the set of filters.

In some embodiments, the new filter is a segment of the selectedcombination.

In some embodiments, any of the new filter and a number of matches ofthe new filter is determined with a randomization process.

In some embodiments, the number of matches is a predetermined quantity.

In some embodiments, the processor is configured to derive multiple newfilters from the selected combination and assign a score to each of themultiple new filters, and add the new filter to the set of filters whenthe score of the new filter is higher than the score of any other newfilter of the multiple new filters.

In some embodiments, the processor is further configured to assign thescore to each filter of the set of filters in accordance with aparameter selected from the group consisting of: respective of applyingthe filter, an expected number of comparisons between the search termand a text; a frequency of occurrence of the filter in the set ofcombinations; a number of matches in the filter; and a span of thefilter.

In some embodiments, the processor is further configured to: create asecond set of filters by: select filters from the set of filters,replace each the selected filters with at least one different filter,the selecting and replacing performed according to a sequence sortedfrom the highest score of the scores of the selected filters to thelowest score of the scores of the selected filter, wherein each one ofthe combinations contains at least one filter from the second set offilters, and select the second set of filters over the sets of filtersin accordance with a fitness score.

In some embodiments, the fitness score is calculated using a parameterselected from the group consisting of: an expected computation speed ofsearch determined as a function of a sum of an expected number ofcomparisons for all the filters in the set of filters; an amount ofmemory required to conduct a search respective of the set of filters.

In some embodiments, the processor is further configured to generatemultiple ones of the set of filters, wherein: each position in thesearch term corresponds to at least one of the multiple ones of the setof filters, wherein each one of the combinations contains, at a positionin the respective one combination, at least one filter from at least oneof the multiple ones of the set of filters, the position of the searchterm corresponding to the position in the respective one combination.i.e. the processor creates multiple sets of filters. Each position inthe search term corresponds to at least one of the sets of filters. Eachcombination contains at least one filter from one of the multiple setsof filters. The position in the combination where the filter is foundcorresponds to the position in the search term that corresponds to theset of filters containing that filter, i.e. the filter in thecombination.

In some embodiments, the processor is configured to form each of themultiple ones of the set of filters by: designating the set ofcombinations as a current set of combination, and designating the set offilter based on the current set of combinations as a current set offilters, obtaining a sequence of positions in the search term, for eachthe position, in accordance with the sequence, generate a set of filtersin association with the position, update a combination from the currentset of combinations when the combination contains a filter from thegenerated set of filters associated with the position and when thefilter is present in the combination at that position, delete a filterfrom the current set of filters when the filter is not retainable withrespect to the current set of combinations, wherein the filter isretainable with respect to the current set of combinations when thereexists at least one combination of the current set of combinations whichcontains only the filter.

In some embodiments, the processor is configured to form the set offilters associated with a position by: obtaining the set ofcombinations, the current set of filters and the position; for eachfilter of the current set of filters: add a filter to the set of filtersassociated with the position when the filter is found in at least one ofthe combinations from the set of combinations at the position, anddelete such combinations from the set of combinations.

In some embodiments, the processor is further configured to: receive alossy threshold, create a subset of the combinations in accordance withthe lossy threshold respective of the search term and the similaritythreshold, wherein generating the set of filters comprises generatingthe filters respective of the subset of the combinations.

In some embodiments, the size of the search term corresponds to multiplepossible sizes for the search term, and wherein the similarity thresholdcorresponds to multiple similarity thresholds, each similarity thresholdcorresponding to a size from the multiple possible sizes for the searchterm.

In some embodiments, the at least one hardware processor is furtherconfigured to form a plurality of sets of sets of filters, where eachone of the plurality of sets of sets of filters corresponds to adifferent sequence of a plurality of sequences of the positions in thesearch term.

In some embodiments, the at least one hardware processor is furtherconfigured to select a set of sets of filters from the plurality of setsof sets of filters according to an expected number of comparisonsperformed when the selected set of sets of filters is applied forexecuting the k-mismatch search, where the expected number ofcomparisons is a function of a given text and a given word for search.

In some embodiments, the at least one hardware processor is furtherconfigured to apply the executing of said k-mismatch search forclustering a plurality of words of a text.

In some embodiments, the at least one hardware processor is furtherconfigured to apply the clustering to accelerate a text indexing step ofthe k-mismatch search.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosed technique will be understood and appreciated more fullyfrom the following detailed description taken in conjunction with thedrawings in which:

FIG. 1 is a flow chart explaining the steps of a method for generating aset of filters;

FIG. 2 is a flow chart explaining the steps of an additional method forfilling a set of filters;

FIG. 3 is an illustration of the use of the set of filters;

FIG. 4 is an illustration of examples of minimal sets of filtersobtained by embodiments of the present invention, their advantage overnaïve algorithm;

FIG. 5A is a flow chart explaining the steps of a method for ak-mismatch search system to create and select a set of filters from aplurality of generated sets of filters;

FIG. 5B is a flow chart explaining the steps of a method for ak-mismatch search system to create and select a set of filters from aplurality of generated sets of filters;

FIG. 6 is a flow chart explaining the steps of for a k-mismatch searchsystem to generate a position-related set of filters:

FIGS. 7A-7C illustrate the generation of a position-associated set offilters; and

FIGS. 8A-8B each illustrate a computer system for executing any of thesteps disclosed herein, constructed and operative in accordance with anembodiment of the disclosed technique.

DETAILED DESCRIPTION

The present invention provides a method for generating an optimal set offilters for a given task of search, taking into account the availablecomputer memory i.e. Random Access Memory (“RAM”) available for thetask.

The present invention may be applied for full or partial calculation ofa Protein Connectivity Network (“PCN”). A PCN is a graph that can beused in order to solve different problems of computational biology,mainly to assist in the prediction of protein structure andfunctionality. The PCN consists of nodes that are small fragments ofprotein sequences, and an edge between nodes reflects high similaritybetween fragments.

Full or partial calculation of the PCN for a large enough proteindatabase is a crucial process, requiring large amount of computationalresources. Therefore, improving the process by an effective algorithm isvery important. The finding of all edges for such graph is an example ofthe k-mismatch search problem.

The task of “k-mismatch problem” is significant for many systems thatsupport word search over large-scale databases. In addition tobioinformatics (i.e. search of DNA and protein subsequences), thepresent invention may be applied for document or internet searches, byrepresenting information such as pictures, sound, video, and3D-structures as linear text, built from a corresponding alphabet.

Embodiments of the present invention may be applied to systems includingdifferent types of large-scale database. For example, a database may bedistributed over multiple servers, or alternatively the database may bemassively partitioned. Since the amount of stored information is growingexponentially, developing an effective algorithm for a quick search onlarge-scale databases is of increased importance.

The term “text” refers hereinafter to digital text which is anelectronic version of written text. The text may be given as a bulk viaelectronic media or accessed remotely via online streaming over anetwork.

The term “similarity” refers hereinafter to a match of a search term inthe searched text or stream of data with specified threshold ofmismatch.

The term “Hamming distance” refers hereinafter to the distance between astring, or alternatively a substring, and a pattern.

Approximate string matching is a technique for finding strings thatapproximately match a pattern, by searching for all the positions in atext where the distance between the sub-string located at that positionand the pattern is smaller or equal to a bound k.

The term “string” refers to herein as any sequence of letters. Thesequence of bits may be a portion of another sequence of bits, e.g. theterm string also includes any substring of a string.

The term “shape” refers hereinafter to a form of a filter. The shapeincludes positions of matches and has a size and span. The size is theamount, or number of matches and the span is the length of the shapeincluding the positions of the matches and the gaps between them if any.For example, the shape {0, 1, 3, 6}, ##-#--#. Its size is 4 and itsshape is 7. The gaps represent positions in which a match may not occur.The sequence of matches in a shape without the gaps is also called seedand gaps are also called spaces.

The term “gapped q-gram filter” refers hereinafter to filters with adefined shape, having the size of q, i.e. the amount or number ofmatches is q. Gapped filters are also called spaced seeds.

The term “filter” may refer hereinafter to a gapped q-gram filter.

The term “indexing” refers hereinafter to a search for all similar termsof each term in a given text.

The term “minimal set of filters” refers hereinafter to a set of filterswhich are a necessary condition for a given threshold of similarity andit may be used for a search term. In other words, there is not anyfilter which can be deleted from the set.

The term “similarity graph of words” refers hereinafter to a graph wherethe nodes are multiplicity of words of a selected size from the text(with overlapping), and the edges reflect the similarity betweenconnected words more or equal than a defined threshold of similarity.

The term “combination” refers herein after to a possible combination ofpositions of matches and gaps meeting the conditions of the similaritythreshold for the given size of a search term, and denoted as a sequenceof ‘0’s and ‘1’s, where, for a given position of the sequence, a ‘0’indicates a mismatch and ‘1’ indicates a match at the correspondingposition.

In general, the term “exhausted” indicates that an iterative process hasrun out of input to continue, whether that input is a set of filters ora set of combinations. For example, the term “the set of allcombinations is exhausted” refers herein after to a situation where theset of all combinations is either empty or completely marked asreviewed.

The term “segment of a combination” refers hereinafter to a sequence ofmatches and mismatches starting from some position of the combination. Afilter that is taken as a segment from a combination begins from amatched position and continues until a predefined number of matches isfound (i.e. at the end of a segment there should be a match). Forexample, for the combination 10011010100111, the filters having 4matches are: 1001101, 110101, 10101001, 1010011 and 100111. Optionally,the number of matches might not be defined, or received as a user input.In such cases the number of matches may be selected randomly from apredefined interval.

A large text (e.g. more than 1 G), may present some limitations on RAMusage. Embodiments of the present invention may be realized in differentcomputational environments, such as distributed over one or morecomputers, collectively having considerable memory resources (e.g.10-100 times larger than the text to be searched), or distributed over anetwork of computers (nodes), each having substantially limited memoryresources (e.g. less or comparable to the size of the text searched.

There disclosed technique offers improvements over memory usage andsearch time over the prior art under a range of conditions: such assearch text content, alphabet, permitted number of mismatches, and wordsize, to name a few.

An exemplary embodiment of the present invention provides a computersystem for generating a set of filters for executing one or morek-mismatch searches over a text. The computer system includes at leastone hardware processor. The computer system may be any of: a distributednetwork of multiple hardware processors; a local network of multiplehardware processors; or a stand-alone computer. The computer systemreceives the following parameters: a size of a search term, a similaritythreshold and a number of matches for a filter. The number of matches inthe filter may be defined automatically, such as on the basis of thesize of the text being searched, and the amount of available RAM.

The computer system generates the set of filters based on a complete setof the possible combinations for positions of matches and gaps for thesize of a search term, and according to the similarity threshold. Forexample, for a search of length 5, having a similarity threshold of 4matches (e.g. at least four out of five bits must match and therefore atmost 1 bit can mismatch), in addition to a perfect match (e.g. zeromismatches) there are 5 possible combinations since the mismatch canoccur at any of positions 1 through 5 of the search term. Thusgenerated, the set of filters may be subsequently reduced to increasethe memory usage and computational efficiency.

More formally, the computer system may repeat the following steps toreduce the size of the set of filters to achieve the minimal set i.e. toyield a minimal set of filters: a) selecting a filter from the set offilters and b) determining the filter as retainable filter if the filteris the only filter from the set of filters that is contained within atleast one of the combinations in the set of combinations. Otherwise,when the filter is not determined a retainable filter, the computersystem removes the filter from the set of filters. Although the set offilters has been reduced to a minimal set, and may still be applied fora search that is not lossy i.e. such that every term is found in thetext.

The memory requirements for implementing the techniques disclosed hereinare determined by the number of filters, among other parameters. Thefollowing description relates to techniques for determining a minimumset of filters, i.e. a sufficient number of filters to cover all thepossible combinations, but no more. The effectiveness of determining theminimal set of filters is determined by the following parameters:

-   -   1) The memory requirements for implementing the mapping step.        This is evaluated as a function of the number of filters in the        minimal filter set (MCS). The complexity of this parameter may        be estimated as O(n*|MCS|), for size of search text n, and        number of filters in the minimal set given by |MCS|.    -   2) The speed of search may be determined according to at least        two factors: the number of comparisons between the search query        and the search text, and the number of “idle” filter comparison.        The latter relates to cases where sub-sequences of a filter are        not found in the search text. For large search texts and a        correspondingly significant number of occurrences of the        filters, the factor largely determining the speed of search        corresponds to the number of comparisons. Conversely, when the        size of the search text is relatively small and the frequency of        occurrences of the filters is correspondently low, then the        limiting factor determining the speed of search is primarily the        number of “idle” checks.

The number of comparisons of the search query with the search text maybe calculated as the sum of the comparisons, as estimated for eachfilter in the MCS, given as follows:

Number of comparisons=P*L*n

Where P is the probability of occurrence for the filter, L is the numberof positions in the search query where the filter is present, and n isthe size of the search text.

The probability of occurrence for the filter, P can be estimated as:

P=1/Σletters in the alphabet^(m)

Where m is the number of matches in the filter.

The number of “idle” checks may be estimated as:

number of idle checks=(1−P)^(n) *L*|MCS|

For L=(size of query)−(span of filter)+1.

Typically, a higher the number of required matches (threshold)corresponds to a larger minimal filter set (MCS). Thus, althoughselecting filters having very low probabilities of occurrence (e.g.having a relatively high_number of required matches) decreases thenumber of comparisons, this requires greater memory resources to accountfor the larger MCS,—and increases the number of “idle” checks. One needsto optimize the selection of the most suitable MCS's for a givenproblem, accounting for memory availability. The optimal number ofmatches in the filters (m), which determines the probability ofoccurrence P and the |MCS| may be determined experimentally, perapplication. To handle situations of insufficient memory, the map of thefull-sized search text may be divided into sections, and the search maybe implemented per section, either serially or in parallel. The size ofeach section, n_(s), may be selected according to the RAM required toperform the mapping for the MCS for that respective section.

For relatively small search texts, or for cases where the search textcan be divided into smaller sections, and searched separately, thefollowing technique may be used:

1) Create multiple MCSs, each of which complies with the given searchrequirements, such as the size of the search query and similaritythreshold (maximal number of mismatches). Arrange the sets according tothe number of matches in the filters. The general rule is: the greaterthe number of matches the fewer the number of comparisons between thesearch query and the search text, demanding more memory (RAM) to performthe mapping, and resulting in a higher number of “idle” checks.

2) Measure speed of the search for each MCS.

3) Select the MCS that provides search results in the shortest amount oftime, i.e. the maximal speed.

The following table illustrates the technique described above forcalculation the minimal filter set (MCS) for a query word of size 10characters. The table illustrates employing the above technique to findall the similar words in the text that contain no more than fourmismatches.

First, all the combinations for matches and mismatches are considered,and a minimal set (MCS) of filters is created such that, eachcombination contains at least one filter from the set. Denoting a matchas “1”, a mismatch (or don't care) as “0”, and:

ABCDEFGHIJ—query1111110000—combination of matches mismatchesABCDEFQQQQ—textTable A below illustrates creating a MCS corresponding to the criteriongiven above, with the filters indicated in bold. When the filter isselected from the combination, one takes the first filter containingfour matches:

TABLE A Combination of matches MCS { } 1111110000 {“1111”} 1111101000{“1111”} 1111011000 {“1111”} 1110111000 {“1111”, “11101”} 1101111000{“1111”, “11101”} 1011111000 {“1111”, “11101”} 1011110100 {“1111”,“11101”} 1011101100 {“1111”, “11101”} 1011011100 {“1111”, “11101”,“101101”} 1010111100 {“1111”, “11101”, “101101”}

Beginning with the null set { }, the first filter is derived from thecombination as the first occurrence of four matches with the searchterm, in this case {“1111” }, and added to the MCS set. This filteroccurs in the next two combinations, and is therefore not added again.The fourth combination does not contain any filter from the current setof filters (e.g. the current set of filters only contains the filter“1111”, which is not found in the fourth combination, “1110111000”).Thus, a new filter is derived to ensure that each combination includes(e.g. is “covered”) by at least one filter. For example, the new filtermay be derived from the uncovered combination by taking the firstsegment of that combination having the number of matches correspondingto the threshold, defining the number of matches in the filters. In theexample above, this segment corresponds to the sequence “11101”, havingfour 1's, corresponding to the threshold of four matches, and positionedat bits 1 through 5 (1 mismatch at bit 4) of the fourth combination.This segment is added as the second filter to the set of filters. Thesenext four combinations include filters that are already included in theMCS, and are thus “covered”. The second last combination provides athird filter: “101101”, which is not yet included in the MCS. Thisfilter is added as the third filter. The last combination includes anoccurrence of a filter that has already been included in the MCS. i.e.“1111”, and is ignored. The result is a MCS with three filters thatcover all the combinations considered in the table.

According to another embodiment of the present invention, the completeset of combinations consists of combinations having a match in the firstposition or, alternatively, in the first and the last position. Suchsets are advantageous in that they may be considerably smaller the fullset, without losing any of the relevant matches.

Exemplary embodiments of the present invention will be explained belowwith reference to the drawings.

FIG. 1 is a flowchart of a method for generating a minimal set offilters 100. The generation of minimal set of filters is a pre-processstage which precedes the search of the search term in a given text. Theset of filters may be reduced by taking into consideration only thenecessary but not sufficient condition, for the word similarity at apredetermined threshold.

The computer system is used to perform the following steps:

An empty set of filters is defined 102. A set of all combinations ofmatches and gaps for each search term of the at least one search termsis generated 104. Due to size limitations of memory, combination may begenerated on the fly and then removed from memory.

After generating a set of all combinations, each combination in the setis examined to see if it contains a filter from the current set offilters 106.

If a filter from the updated set of filters is not found in thecombination, a filter is derived from the combination and the newlyderived filter is added to the set of filters 108. The newly derivedfilter may be a segment of the combination or may be randomly derived.If the examined combination contains at least one filter from the set offilters, the next combination is examined. The size of the filter may bebased on the number of matches, which may be received as an input.Alternatively, the number of matches may be randomly selected. Thesegment of the combination may also be referred to as shape.

The size of memory required for the pre-processed text is proportionalto the size of the text being searched, multiplied by the size of theset of filters. If the size of the pre-processed text is too large, thena smaller shape size, i.e. smaller number of matches may be considered.The greater the number of matches, the larger the set of filters, whichincreases the amount of memory required for the pre-processed textstage.

Alternatively, the text may be divided into smaller parts and each partmay be processed separately in a parallel computing environment (e.g.two or more computer processors may be used simultaneously).

After all the combinations from the set of all combinations are checked,the set of filters is reduced according to the following steps: Eachfilter in the set of filters is examined 110, by examining eachcombination in the set of all combinations 112 to verify that the filteris a singular occurrence in the combination. A singular occurrence meansthat although a combination may contain multiple occurrences of the samefilter, if there is an occurrence of a different filter from the set offilters in all the combinations, the examined filter is not a singularoccurrence. If the examined filter is not a singular occurrence, it isdeleted from the set of filters 114. Then, the next filter in theupdated set of filters is examined until all the current filters in theset of filters have been examined. The resulting set of filters includesonly filters that are a singular occurrence in at least one of thecombinations. This is the minimal set of filters 116.

Embodiments of the present invention provide detailed steps for creatingthe set of filters, as described above in FIG. 1 in steps 106-108. Analternative method to these steps is detailed in FIG. 2 and described asfollows. All occurrences of filters from each combination are identified202. If the number of all the possible filters is too large, it can belimited by additional conditions, such as a fixed upper bound ofmatches, a maximal span, the selection of filters may be limited to be asegment from a combination, to name but a few possible limitations. Aweight, or score is assigned to each filter. For example the score isdetermined according to the sum total of combinations where the filteroccurred 204. The filter with the highest score is added to the set offilters 206. The combinations containing this filter may be marked asreviewed 208 or removed from the set of combinations that are beingexamined. Process 200 i.e. steps 202-208 may be repeated for anyunmarked combinations that have not been removed, until a good set offilters is found. A “good” set of filters is a set of filters that maybe used to conduct a faster search or yield higher quality results ascompared to prior art methods. For some search conditions, executingprocess 200 provides an effective set of filters in a shorter amount oftime as compared to prior art methods. To generate a minimal set offilters, steps 202-206 may be repeated for any unmarked (or not removed)combinations until a satisfactory filter set has been built, or the setof the combinations is empty or is completely marked as reviewed.

With reference to FIG. 3, an exemplary illustration of the use of theset of filters is shown. For a search word, W=“AARRCCDDEE”, and an MCS,for S={XXX, XX-X}, corresponding to two filters. The first filtercorresponding to three sequential matches (XXX), and the second filtercorresponding to two sequential matches, followed by a mismatch, and athird match (XX-X). The extracted set S_(w) includes the 15 words listedin table 300, of which the 12^(th) word is RC-D. Given a portion of atext: “GGGGGGGRCGDGGGGGGGGG”, this portion contains only one instance ofthe word RC-D, corresponding to the configuration XX-X, and does notcontain the other words. Thus, the comparison with the word RC-D occursin one position only, with the character “G” in the text aligning withthe character “C” in the search word, resulting in one mismatch:

GGGGGGG RC G D GGGGGGGGG     •••||•|•••     AAR RC C D DEE

According to another embodiment of the present invention, the set offilters may not be generated or calculated for each user but rather aset of filters may be calculated for a wide range of parameters andstored in an accessible public database.

Reference is now made to FIG. 4, which is an illustration of severalexemplary minimal sets of filters obtained by embodiments of the presentinvention. The advantage of these minimal sets over the naïve algorithmis described below.

The size of word |W|=20, the similarity threshold T=60% (i.e. up to 8mismatches are permitted). |S|=size of the minimal set of filters S,n_(C)=number of produced words from a term for search in all filters ofS, M=number of matches in a filter. The advantage (A) in comparison tothe naïve algorithm is calculated as d^(M)/n_(C), where d=the size ofthe alphabet, d=20.

According to another embodiment of the disclosed technique, a heuristicapproach is used to find a good set of filters. The heuristic approachis applied by forming a plurality of sets of filters, according to apredefined order and assigning a score to each set of filters. The scoremay be assigned to each set of filters based on (i) its size, i.e.amount of filters in the set; and/or (ii) effectiveness, i.e. amount ofwords generated from that set of filters. Generally, the length of thefilter influences the number of words that are generated from it. Thatis, as the length of the filter increases, the number of wordsdecreases, thereby increasing the speed of the search.

Referring to FIG. 5A, a good set of filters may be obtained by comparingthe score assigned to each of the multiple sets of filters. Each set offilters may be formed after first performing the following two actions:i) generating a complete set of filters and attributing a weight, orscore to each filter 502 and ii) sorting the filters according to theassigned score 504. The sorting may be in descending order.

Each set of filters 506 may be formed by the following steps: The set offilters is generated, and initiated. If it is not the first set offilters, one or more filters from the complete set of filters 508 areignored. The filter with the highest score is added to the generated setof filters. Any combinations containing the filter with the highestscore is marked as reviewed, and may be removed from the set ofcombinations to be examined. The above is repeated for unmarkedcombinations that have not been removed, until a satisfactory filter sethas been built, or until the set of the combinations is empty or the setof combinations is exhausted (all marked as reviewed) 510. Allnon-retainable filters are removed from the set of filters, and afitness score is assigned to the set of filters according to one or morepredefined criteria 512. The set of filters with the best fitness scoreis retained 514. The best fitness score may be the highest score or thelowest score depending on the optimization criteria. The above steps forforming a set of filters may be repeated until the ignoring one or morefilters of step 508 comprises deleting all the filters, excluding thefilter with the highest score. Under some conditions, the cycle mightnot terminate due to the high number of possible combinations of filtersthat can be ignored. In such cases the steps may be repeated accordingto available computer power and time to obtain a satisfactory solution.

The predefined criteria for scoring each set of filters may be thenumber of filters in the set of filters and the number of wordsgenerated from the set of filters for a given search term.

The criterion for inclusion/preclusion of any of the one or more filtersin 508 may be performed according to an element in a predefined seriesof arithmetic sequences. The predefined series of arithmetic sequencesmay be such that each element in the series is an arithmetic sequencewith a common difference of d. In a non-limiting example, forming of thesecond set of filters may preclude the second highest filter in step508, the forming of the third set of filters may preclude the second andthird highest filters in step 508, the forming of the fourth set offilters may preclude the second, third and fourth highest filters instep 508 and so on. Therefore, according to this example d=1 and theelements in the series of arithmetic sequences are ({2}, {2, 3}, {2, 3,4} etc.).

Reference is now made to FIG. 5B which illustrates a heuristic approachfor finding a satisfactory set of filters, constructed and operative inaccordance with another embodiment of the disclosed technique. Thecomputer system performs the following steps:

A complete set of filters C is generated.

-   -   i. A weight (or score) is assigned to each filter; and    -   ii. the filters are sorted in descending order according to        their respective weights (502 b). For each filter i in C        (starting from the most frequent i.e. highest weight) (504 b)        -   a. filters in C (which are less frequent than filter i) are            marked according to a selected order, (detailed below) 506            b.        -   b. A minimal set of filters is created according to the            method illustrated in FIG. 2, Steps 206-208 are repeated for            unmarked filters, starting from i 508 b.        -   c. The effectiveness of the minimal set of filters is            calculated.        -   d. If the effectiveness is better than the effectiveness for            the best minimal set of filters, the minimal set of filters            is reserved as the best minimal set of filters 510 b.        -   e. The filters in C are unmarked, and the next marked filter            is examined 512 b.        -   f. The above steps are repeated for the next filter i 514 b.            The output of these steps is the best minimal set of filters            516 b.

Reference is now made to FIG. 6, which illustrates a heuristic approachfor decreasing the number of generated words from a set of filters,constructed and operative in accordance with another embodiment of thedisclosed technique. Decreasing the number of generated words from a setof filters may increase the speed of the search. The approach builds oneor more filters for each position of the search word, taking intoaccount the fact that there is a correlation of presence or absence ofthe filter in different positions in a word. Accordingly,position-related sets of filters are formed and each set of filters isassociated to a particular position in a given search term. The filtersmay be of different sizes. One optimization disclosed is to only includefilters that are actually found respective of a specific position in thesearch word.

The computer system performs the following steps: A set of filters (MCS)602 is formed (by, for example, implementing one of the algorithmsdescribed hereinabove, to create a set of filters that covers all thepossible combinations). The current position is set as the firstposition 604. The formed set of filters is associated with the currentposition 606. The first position may be the most left position of agiven search term or may be ‘low-occurrence’ positions as detailedbelow. Alternatively, some sequence S of positions is selected in thesearched term (the simplest sequence can be S={(1, 2, . . . , n−k+1}(k−the span of the smallest filter). According to the present invention,any other order may be used, and the first position in S is set as acurrent position 604.

All combinations in the set of all combinations containing a filter atthe current position are removed, or marking as reviewed 608. The nextposition in S is set as the current position 610. The current positionmay be shifted by 1 to the right. The size of the set of filters isreduced to minimum i.e. to yield a minimal set of filters, after step610 as described above. Any filter which is not determined as aretainable filter in the set of not removed, or not marked combinationsis removed, or marking as reviewed, and the updated set filters is setas the current set of filters 611. The steps 606 through 611 may berepeated until the set of the combinations is exhausted. The obtainedset of filters is the largest for the first position of the word,decreasing from the left position to the right in the sequence S.

The output of the method described in FIG. 6 is a set of sets. Each setof the set is associated with some position in the search term. One set(associated with the starting position) is largest (and the forms ofthis set should be used for mapping).

The effectiveness of the position related sets of filters may becalculated as the sum of comparisons of the filters associated with eachposition (n_(C)). The application of the position related set of filtersmay be performed by the following steps; mapping all words in allfilters in the text; extracting for each position of term all wordsaccording to the correspondent associated with this position set offilters; comparing the terms with corresponding places in the textaccording to the mapping of the words extracted.

Table B below illustrates the results of scoring a set of filters. Thescores are assigned according to the number of words generated from theset of filters for a given search term. Consider the first set in FIG.4: S={XXX, XX-X}. The set was found for search term size |W|=20, and thesimilarity threshold T=60% (i.e. up to 8 mismatches are permitted). Thenumber of words generated from the set of filters for a given searchterm is 35. Consider a search term: “abcdefghijklmnopqrst”. Thefollowing Table shows that for the filter XXX, the words will be:

TABLE B  abCdefghijklmnopqrSt abCdefghijklmnopqrSt 1 abC 1 ab-d 2 bCd 2bC-e 3 Cde 3 Cd-f 4 def 4 de-g 5 efg 5 ef-h 6 fgh 6 fg-i 7 ghi 7 gh-j 8hij 8 hi-k 9 ijk 9 ij-l 10 jkl 10 jk-m 11 klm 11 kl-n 12 lmn 12 lm-o 13mno 13 mn-p 14 nop 14 no-q 15 opq 15 op-r 16 pqr 16 pq-S 17 qrs 17 qr-t18 rstThe number of words n_(C(i)) for each filter can be calculated as:

n _(C(i)) =|W|−|C(i)|+1

where: |W|=size of the search term W, |C(i)|=span of filter i, includingspaces.

Reference is now made to FIGS. 7A-7C which shows severalposition-related sets of filters, constructed and operative inaccordance with another embodiment of the disclosed technique. Theeffectiveness of the algorithm may be significantly increased by usingposition related sets of filters suitable to a specified search term.This may account for occurrences of each extracted word in a formedposition related set of filters in the text. Thus, the ‘low-occurrence’positions of the specified search term may be associated with thelargest sets, while the ‘high-occurrence’ positions may be associatedwith the smallest sets. The sets of filters were calculated for thesearch conditions described in FIG. 4 (search term size |W|=20, and thesimilarity threshold T=60% (i.e. up to 8 mismatches are permitted). Thesequence of the positions is S={1, 2, 3, . . . }. The starting sets aresets given in the FIG. 4.

Referring to FIG. 7A, an example of the position-related set of filtersis shown for the number of matches in the filter, M=3, corresponding tothe top row of Table 400 in FIG. 4. This means one should makecomparisons with the form “XXX” from the search word only up to the 13thposition. Referring to FIG. 713, an example of the position-related setof filters is shown for the number of matches in the filter, M=4,corresponding to the second top row of Table 400 in FIG. 4. Referring toTable C in FIG. 7C, the scores of the position-associated sets arecompared with the score for the filter sets presented in FIG. 4, wherethe scores are determined according to the number of words generatedfrom the set of filters for a given search term.

According to another exemplary embodiment of the present invention, anarray of sets of sets may be prepared, as follows: for a given searchword, starting from different positions, a lowest occurrence position isselected. The set of sets that start from this lowest occurrenceposition is used. Similarly, the array of set of sets can contain a setof sets having different assignments for the next positions in S. In thelatter case a first and second lowest occurrence position may beselected, and the set of sets with corresponding first and second lowestoccurrence positions are used. The procedure continues for largernumbers of positions.

The ‘low-occurrence’ position is calculated as the sum of occurrences ofall words starting from this position for all filters from the set offilters associated with this position. The word occurrences may beprovided from the mapping of all words for all filters from the set offilters in the text (text pre-proceeding).

Embodiments of the present invention may be adopted for severalimplementations: (i) searching of all similar words for one word or aset of words in a given text; (ii) searching of all similar words forone word or a set of words in an online stream; and (iii) searching foreach word in a first text similar words in a second text; (iv) indexing,i.e. searching for each word of a text similar words in the same text(which may be considered as building of similarity graph of words of thegiven text).

Usage of full set of all filters for search is described in Frenkel, Z.M. and Trifonov, E. N. Evolutionary networks in the formatted proteinsequence space. Journal of Computational Biology, 14(8):1044-57, October2007. The full set of all filters is generated by exhaustivecombinatorics.

An exemplary embodiment of the present invention provides a method forsearching for the words that are similar to one word or a set of wordsin a given text, using the minimal set of filters described above,thereby providing an improvement over the exhaustive approach. Textindexing is similar to constructing a similarity graph for the text. Thesimilarity graph is characterized by a plurality of nodes connected by aplurality of edges. The nodes represent a plurality of repetitions ofcharacter sequences, corresponding to the size of the search term; eachedge connects a pair of nodes is based on the respective similaritythreshold between the nodes. The similarity graph may be compressedaccording to conventional methods, such as the method described in[PCT/IL2016/051220]. The clustering of the words may significantlyreduce the time needed to complete a search.

This may be explained as follows. Consider the following Lemma: If thefollowing conditions are satisfied:

-   -   1. The words A, B, C, D are each of size m    -   2. The word A is similar to the word B, having a maximum number        of mismatches x, i.e. the distance d between A and B is (d(A,        B)≤x)    -   3. The word C is similar to the word D, having a maximum number        of mismatches y i.e. the distance d between C and D is (d(C,        D)≤y)    -   4. The word A is different from the word C, having a maximum        number of matches z, i.e. at least m−z mismatches. Thus the        distance d between A and C is (d(A, C)≥m−z)        Then, word B is similar to D within a maximum number of        mismatches given by: max{0; m−(x+y+z)}, and the following        relationship holds:

d(B,D)≤max{0;m−(x+y+z)}

The proof of this result is as follows. One can consider two circles,the first circle has radius x with A in the center and B on the edge.The second circle has radius y, with C in the center and D on the edge.The circles can overlap for the case of the minimal distance between Aand C, or the circles can be spread apart for the case of the maximumdistance between A and C The two possible cases are: 1) the minimaldistance between B and D is given by: m−(x+y+z), and 2) the minimaldistance between B and D is 0 (i.e. the circles overlap).

The above lemma may be applied to performing a search, as follows. Inorder to check if two clusters match within 60% similarity, (i.e. for asequence of 20 characters, up to eight mismatches, or at least 12matches), the clusters are built to comply with a similarity criterion.For example, one such criterion may be that all the sequences in eachcluster are similar to the correspondent cluster-seeding sequence withinat least a 90% similarity measure (i.e. maximum two mismatches).

If each of the seeding sequences have a maximum similarity of 35% (i.e.at most 7 matches) then the maximum similarity between any two sequencesof the two clusters cannot exceed 11 matches (i.e. 55%). In other words,the minimal distance between the clusters is at least 9 mismatches(20−(7+2+2)=9). In such a case, a connection complying with a 60%similarity requirement between the two clusters is impossible. Thus,such pairs of clusters need not be checked, to optimize memory, time andcomputational resources. If the first cluster has m sequences and thesecond cluster has n sequences, the economy is m*n−1 comparisons. It maybe noted, that the probability of finding a sequence fragment of 35% ofsimilarity in the database is relatively low (about 10⁻⁵ for 20-letteralphabet).

The following indexing algorithm may be applied, for th1 representing athreshold for the minimal number of matches needed for indexing, and forth2 representing the maximal number of mismatches needed for clustering:

1) Divide the text into a set of clusters of words. The clusters areformed as follow:

a) For each word that is not in cluster:

b) find all similar words at the selected threshold th2, i.e. by usingthe k-mismatch search described above;

c) Sort the words in the cluster: the words more similar to the seedword are assigned smaller indices;

2) Check that the connections between the clusters comply with thefollowing condition(s): the cluster are connected if there is at leastone word from the first cluster that is similar to at least one wordfrom the second cluster with the similarity threshold, th1:a) For the seed words (determined according to the k-mismatch searchtechnique described above), find all the seed words similar to within atleast (K1=th1−2*th2) matches. i.e. similar words can only appear in suchclusters;b) Compare only words of indices i and j of the clusters, whereK₁+K_(i)+K_(j)≥th1 (i.e. K_(t) is the number of matches between the seedwords, K_(i) and K_(j) are the number of mismatches of the words i andj, respectively, with the corresponding seed words).c) if a connection is detected, the procedure terminates.3) Complete the indexing. This step is optional since for manyapplications, identifying a connection between clusters is sufficient.a) compare all the nodes within each cluster;b) compare the words of connected clusters (similarly, only words ofthat indexes i and j of the clusters, where K₁+K_(i)+K_(j)≥th1, whereK₁, K_(i), and K_(j) are as above.

The text indexing algorithm is now given as follows: i) all the wordspresent in all filters are mapped in the text. ii) for each search word,all the words in all filters are extracted, and the terms are comparedwith the search text according to the mapping of the extracted words.For fast access, addresses of the extracted words may be used.

According to another exemplary embodiment of the present invention,under certain conditions, the words in all filters in the search termsare mapped and stored in a suffix tree data structure for a set ofsearch terms. The comparison with the whole set of filters is performedin linear time. Such conditions may include when the pre-processing ofthe text cannot be implemented. For example: when the text isunavailable, or the text streams on-line or the amount of availablememory is not enough.

In one exemplary embodiment, ‘low-occurrence’ positions of the specifiedsearch term may be associated with largest sets, while the‘high-occurrence’ positions may be associated with the smallest sets.Thus two position-associated sets of filters are calculated for twoorders of positions S1 and S2.

Referring to Table C below, two position-associated sets of filters areshown for two possible sequences, of positions, S1, S2, in the searchterm, where XXX refers to three sequential matches, and XX-X refers totwo sequential matches, follows by one wild card, and one match:

TABLE C S₁ = { 1, 2, 3, 4, . . . , S₂ = { 14, 15, 3, 4, . . . , 15, 16,17, 18 } 12, 13, 1, 2, 16, 17, 18 } Position Set Position Set 1 XXX,XX-X 1 XXX, XX-X 2 XXX, XXX 2 XX-X . . . . . . 3 XXX, XX-X 12 XXX, XX--X4 XXX, XX-X 13 XXX, XX-X . . . . . . 14 XX-X 14 XXX, XX-X 15 XX-X 15XXX, XX-X 16 XX-X 16 XX-X 17 XX-X 17 XX-X 18 — 18 —

Consider a word for search W=‘ABCDEFGHIJKLMNOPQRST’. Suppose occurrencesof all three-letter words in the text are the same—equal to zero,excluding ‘BCD’ (i.e. XXX) and ‘BC-E’ (i.e. XX-X), which are equal to1000 occurrences each. To find all words similar to the search word W inthe text (with 60% similarity threshold), according to the left set ofsets (calculated with S1), one would use the sequences ‘BCD’ and ‘BC-E’,which requires performing 2000 comparisons, whereas finding all thewords similar to W with 60% similarity using the right set of sets(calculated with S2), where ‘BC-E’ is not used, will require performing1000 comparisons. Thus the right set is more efficient for such a searchword and such a text.

When the positional associated set of the sets of filters is prepared asdescribed herein, the resulting positional associated set depends on theorder or “sequence” of the positions in the search term, such asillustrated above in Table C). Although the expected number ofcomparisons may be the same from using a different set of sets offilter, the number of filters corresponding to a specific position maybe substantially different, affecting performance. After mapping thefilters in a specific text for search, the occurrences of the searchwords, in the form of all the filters, is known, and can be used tofurther optimize the search. For example, occurrences of ‘BCD’, ‘BC-E’in the text for search, as well as any other words from the search termin the example above are known. Thus, for a given search word, thecorresponding set of sets that provide the minimal number of comparisonswith the text may be selected to improve the speed of search. Thesequence of positions that provides the set of sets having the minimalof comparisons with the search term is selected. However, the “general”set of filters, or initial set of filters for all the set of sets offilters is the same. This allows using the same mapping for all possiblepositional set of sets of filters.

According to yet another exemplary embodiment of the present invention,searching for a word or more may be performed by initially constructinga similarity graph of the given text. The similarity graph may becompressed according to a higher similarity threshold where theconnection between the “super-nodes” indicates that there is at leastone connection between the corresponding subgraphs in the primary graph.The minimal set of filters may be applied in the process of finding theconnection between the super-nodes.

The following is an example of using a compressed similarity graph forindexing:

Task: for the text T build the similarity graph, i.e. find for each wordall words similar at least with a threshold thr1.

-   -   i. Create clusters (with selected maximal size) as subgraphs,        where the condition for connection is a threshold thr2, which is        higher than thr1. For example, is thr1 is 60% matches, thr2 can        be 80-90% of matches. It can be demonstrated, that required much        less set of filters, that is required much less time and RAM for        calculation.    -   ii. Calculate connections between the clusters (which is also        much quicker than full network calculation. That means, if there        are at least one pair of nodes, first from the one cluster and        second from another cluster, which are connected (with thr1),        the clusters called “connected”.    -   iii. Calculate the similarity graph on the basis of the        calculated connected clusters. This task is also much quicker        than the direct calculation (because the comparisons should be        carried out only inside the clusters and between the connected        clusters.

According to yet another exemplary embodiment of the present invention,the k-mismatch search may be lossy, in the sense that the matching wordwill be found in the text within a probability threshold. The algorithmdescribed above is “lossless”, i.e. it does not lose any words that aresimilar to the search word within a selected threshold if the consideredset of the combinations contains all possible combinations of thepositions of matches and gaps for given threshold and search term size.However, the disclosed technique may also be used as a “lossy”algorithm, i.e. to find the similar words within at least somepredefined probability. One advantage of this approach is the increasedspeed of search and decreased RAM requirements for the pre-proceeding.

To adapt the technique for a “lossy” algorithm, a set of filters isconstructed that cover only part of the full set, i.e. a subset, of thecomplete set of combinations of positions of matches and gaps. Theproportion of the subset size to the full set should be greater than orequal to the predefined probability of finding the search word. In thiscase the algorithm is repeated until the subset of the full set of acomplete set of combinations of positions of matches and gaps is bemarked or removed, as opposed to performing this until the entire set ofcombinations is exhausted.

In some applications the search term may not limited by a selected size(with a corresponding number of permitted mismatches). Rather, thesearch term may have a permitted size (S) that may vary, for examplebetween a predefined lower boundary (S_(L)) to some predefined upperboundary (S_(U)). The number of permitted mismatches may also be varybetween high a low mismatch thresholds. For example the number ofpermitted mismatches may be proportional to the size of the search term,the context of the application, characteristics of the text, to name buta few. In such cases it may be useful to create a “universal” set offilters that may be applied to a variety of different contexts within arange of search word size and/or number of mismatches.

Although for a permitted search term size S, a corresponding set offilters is required to provide the optimal speed of search, the mapping,which requires RAM resources, does not have to be repeated for eachdistinct size within the range of permissible sizes for S. Animplementation for an algorithm to calculate such that a “universal” setof filters is given as follow:

-   -   i. Construct the set of filters (by one of the above algorithm)        for S_(L);    -   ii. Add this set to set of filters for mapping (SFM);    -   iii. For each S_(i) from S_(L+1) to S_(u) build the set of        filters by, for example, algorithm described in Paragraph 1 in        the claims, using the SFM as an initial set (i.e., instead of        empty initial set used in the algorithm).    -   iv. If new filters are added to the set, checking “if the filter        is retainable” should be done starting from the new filters.    -   v. Add new (retainable) filters to the SFM.

As result, a set of set of filters is constructed containing a set foreach possible size of the search term (with corresponding similarityconditions). However, only filters from the SFM are present in thesesets. Thus, only these filters should be mapped in the pre-proceeding.

For the cases when the size of the search term |W| is larger than S_(U),the search term may be divided into two words of equal size W_(left) andW_(right), for each word all similar words (with the same similaritythreshold (in %)) should be detected in the text. Upstream text of thewords similar to W_(right) should be compared with W_(left), anddownstream text of the words similar to W_(left) should be compared withW_(right).

The techniques described herein may be applied to any number ofcontexts. For example, when searching for words in a dictionary, oftenthe search term must undergo a “normalization” stage, to account fordiscrepancies in spelling, tense, single/plural, etc. Thus thetechniques disclosed herein may be used in place of such a normalizationstage, or in conjunction with a normalization to provide better results.

The set of filters obtained as described above may be adapted forsearching in a list of words (dictionary). In this case, a position ofeach word in each filter should be taken into account in the mapping.Only those words of the dictionary that contain a corresponding filterat the same position to the search term should be selected forcomparison with the search term.

Additional techniques as are known in the art may be used in conjunctionwith the system and methods disclosed herein, to enhance performance.Under certain conditions the size of the alphabet affects theperformance, or effectiveness of the algorithm. Smaller alphabets mayreduce the combinatoric complexity of the search, thus in some contextsit may be possible to cluster two or more letters to one letter, toreduce the size of the alphabet, increasing the effectiveness of thealgorithm. One possible substitution may be to cluster two or moreletters from the initial alphabet to one letter of the new alphabet.Alternatively, in some cases, increasing the size of the alphabet maydecrease the number of occurrences of each filter. For example, the textfor DNA has a 4-letter alphabet (a, t, c, g), corresponding to the fournucleotides present in DNA. The probability of occurrence of anythree-letter words is thus ¼*¼*¼= 1/64. A new alphabet of size 16 (=4²)may be built from all the two-letter combinations as follows: {aa, at,ac, ag, ta, tt, tc, tg, . . . }. The probability of occurrence of anythree-letter word will now be 1/16* 1/16* 1/16= 1/4096, i.e.considerably smaller. This may increase the efficiency of the searchconsiderably. The alphabet size may be increased or decreasedaccordingly.

Additional implementations: a database for filters may be created fordifferent applications. Genetic algorithms may be used to improve thequality (effectiveness) of the set of filters.

Reference is now made to FIG. 8A which illustrates a computer system 800for executing any of the steps disclosed herein, constructed andoperative in accordance with an embodiment of the disclosed technique.Computer system 800 includes at least one processor 802 (e.g. a CPU), arandom access memory (RAM) unit 804, a disk storage unit 806, atransceiver 808, and a power supply 810. At least one processor 802, RAMunit 804, disk storage unit 806 and transceiver 808 are electricallycoupled to power supply 810. RAM unit 804, disk storage unit 806 andtransceiver 808 are communicatively coupled to at least one processor802. Transceiver 808 is operative to send and receive data, comprisingat least one or more program code instructions, and data pertaining tothe k-mismatch search technique disclosed herein. Transceiver 808 maycommunicate with additional computer systems (not shown) using anysuitable wired or wireless protocol, such as via electrical or opticalcable, short and medium range communications means (e.g. Bluetooth.Wifi, IR), as well as longer range (e.g. RF and cellular), to name afew.

RAM unit 804 and disk storage unit 806 are each operative to store oneor more program code instructions and data pertaining to the k-mismatchsearch technique disclosed herein, such as any may be received viatransceiver 808. At least one processor 802 is operative to read the oneor more program code instructions and data pertaining to the k-mismatchsearch technique, and execute the program code instructions on the data,thereby execute any of the steps described herein. At least oneprocessor 802 is operative to write the results of the execution to RAMunit 804 and to disk storage unit 806. At least one processor 802 isoperative to request data and provide data from and to additionalcomputer system (not shown) via transceiver 808. At least one processor802 is operative to provide the results of the execution to a user, suchas via a user interface (not shown).

Reference is now made to FIG. 8B, while still referring to FIG. 8A,which illustrates a networked computer system 820 for executing any ofthe steps disclosed herein above, constructed and operative inaccordance with another embodiment of the disclosed technique. Computersystem 820 includes multiple nodes 800A, 800B, . . . 800N, and acommunications network 822. Each of nodes 800A, 800B, . . . 800Ncorrespond to computer system 800 of FIG. 8A. Thus, each of nodes 800A,800B, . . . 800N include at least the components referred to above withrespect to FIG. 8A, e.g. at least one processor 802 (e.g. a CPU), arandom access memory (RAM) unit 804, a disk storage unit 806, atransceiver 808, and a power supply 810. Nodes 800A, 800B, . . . 800Nare operative to communicate with each other via communications network822 and respective transceivers 808 of each of nodes 800A, 800B, . . .800N. Computer system 820 is operative to execute any of the stepsdescribe herein in a distributed manner over nodes 800A, 800B, . . .800N, using conventional techniques, as are known in the art ofdistributed computing. The steps may be performed in any combination ofparallel and serial computing.

Computer system 800 and computer system 820 are each configured toperform at least the following steps:

A size of a search term and a similarity threshold are received. A setof filter is generated. A set of filters is formed based on a set ofcombinations of positions of matches and gaps for the size of the searchterm according to the similarity threshold. Each one of the combinationscontains at least one filter from the set of filters. In someembodiments, each combination of the set of combinations includes amatch at the first position of the respective combination. In someembodiments, each set of combinations additionally includes a match atthe last position of the respective combination. With reference to FIG.8A, processor 802 receives the search term and the similarity threshold,such as via transceiver 808. Processor 802 forms a set of filters basedon a set of combinations of positions of matches and gaps for the sizeof the search term according to the similarity threshold. Each one ofthe combinations contains at least one filter from the set of filters.

For each filter of the set of filters, it is determined that therespective filter is a retainable filter when there exists at least onecombination of the set of combinations, which contains only that filter.The filter is deleted from the set of filters when the filter is not aretainable filter. With reference to FIG. 8A, processor 802 determinesthat the filter is a retainable filter when at least one combination ofthe set of combinations contains only that filter. If processor 802determines that a filter is not a retainable filter, processor 802deletes the filter from the set of filters.

The filters are formed according to the following steps: The set offilters is initialized. For each selected combination of the set ofcombinations: an occurrence of a filter of the set of filters issearched for in the selected combination; a new filter is derived fromthe selected combination when the occurrence of a filter is not found inthe selected combination. The new filter is added to the set of filters.In some embodiments, the new filter is a segment of the selectedcombination. In some embodiments, any of the new filter and a number ofmatches of the new filter is determined with a randomization process. Insome embodiments, the number of matches is a predetermined quantity. Insome embodiments, multiple new filters are derived from the selectedcombination and a score is assigned to each of the multiple new filters.The new filter is added to the set of filters when the score of the newfilter is higher than the score of any other new filter of the multiplenew filters. In some embodiments, the score may be assigned to eachfilter in accordance with any combination of the following parameters:an expected number of comparisons between the search term and a textrespective of applying the filter; a frequency of occurrence of thefilter in the set of combinations; a number of matches in the filter;and a span of the filter. With reference to FIG. 8A, processor 802 formsa set of filters as follows: processor 802 initializes the set offilters. For each selected combination of the set of combinations:processor 802 searches, in the selected combination, for an occurrenceof a filter of the set of filters. If processor 802 does not find anoccurrence of the filter in the selected combination, processor 802derives a new filter from the selected combination, and adds the newfilter to the set of filters.

A second set of filters may be created according to the following steps:one or more filters may be selected from the set of filters. Each of theselected filters may be replaced with at least one different filter,where the selection and replacements are performed according to asequence sorted from the highest score of the selected filters to thelowest score of the selected filter, where each one of the combinationsmay contain at least one filter from the second set of filters. Thesecond set of filters is selected over the set of filters in accordancewith a fitness score. The fitness score may be calculated using one ormore of the following parameters: an expected computation speed ofsearch determined as a function of a sum of an expected number ofcomparisons for all the filters in the set of filters; and an amount ofmemory required to conduct a search respective of the set of filters.With reference to FIG. 8A, processor 802 creates the second set offilters according to the steps described hereinabove. Processor 802selects the second set of filters of the original set of filters inaccordance with a fitness score.

Multiple sets of filters may be generated as follows. Each position inthe search term corresponds to at least one of the multiple sets offilters. Each one of the combinations contains, at a position in therespective combination, at least one filter from at least one of themultiple sets of filters. The position of the search term correspondingto the at least one of the multiple sets of filters corresponds to theposition in the respective combination. With reference to FIG. 8A,processor 802 generates multiple sets of filters according to the stepsdescribed hereinabove.

Each of the multiple sets of filters is formed as follows: The set ofcombinations is designated as a current set of combination, and the setof filter based on the current set of combinations is set as the currentset of filters. A sequence of positions in the search term is obtained.Following according to the sequence, for each position, a set of filtersis generated in association with the position. A combination from thecurrent set of combinations is updated if the combination contains afilter from the generated set of filters that is associated with theposition. With reference to FIG. 8A, processor 802 creates multiple setsof filters. Each position in the search term corresponds to at least oneof the sets of filters. Each combination contains at least one filterfrom one of the multiple sets of filters. The position in thecombination where the filter is found corresponds to the position in thesearch term that corresponds to the set of filters containing thatfilter, i.e. the filter in the combination.

A filter is deleted from the current set of filters when the filter isnot retainable with respect to the current set of combinations, where afilter is retainable with respect to the current set of combinationswhen there exists at least one combination of the current set ofcombinations which contains only that filter. With reference to FIG. 8A,processor 802 deletes a filter from the current set of filters when thefilter is not retainable.

The set of filters associated with a position is formed as follows: Theset of combinations is obtained, as well as the current set of filtersand the position. For each filter of the current set of filters: thefilter is added to the set of filters associated with the position whenthe filter is found in at least one of the combinations from the set ofcombinations, at the position, and, accordingly, the at least onecombination is deleted, removed, or otherwise marked, from the set ofcombinations. With reference to FIG. 8A, processor 802 associates theset of filter with a position according to the steps describedhereinabove.

In some embodiments, a plurality of sets of sets of filters is formed,where each one of the plurality of sets of sets of filters correspondsto a different sequence of a plurality of sequences of the positions inthe search term. The set of sets of filters may be selected from themultiple sets of sets of filters according to an expected number ofcomparisons performed when the selected set of sets of filters isapplied for executing the k-mismatch search. The expected number ofcomparisons is a function of a given text and given word for search.

In some embodiments, a lossy threshold is received. A subset of thecombinations is created in accordance with the lossy thresholdrespective of the search term and the similarity threshold, wheregenerating the set of filters comprises generating the filtersrespective of the subset of combinations. With reference to FIG. 8A,processor 802 receives the lossy threshold, such as via transceiver 808.

In some embodiments, the size of the search term corresponds to multiplepossible sizes for the search term, and the similarity thresholdcorresponds to multiple similarity thresholds, each similarity thresholdcorresponding to a size from the multiple possible sizes for the searchterm.

In some embodiments, the execution of the k-mismatch search is appliedfor clustering a plurality of words of a text. The clustering may beapplied to accelerate a text indexing step of the k-mismatch search.

The present invention is not limited to the embodiments described above,but encompasses any and all embodiments within the scope of thefollowing claims. It will be appreciated by persons skilled in the artthat the disclosed technique is not limited to what has beenparticularly shown and described hereinabove. Rather the scope of thedisclosed technique is defined only by the claims, possiblerepresentative approximations of which follow:

What is claimed is:
 1. A computer system for executing k-mismatchsearches, comprising at least one processor, configured to: receive asize of a search term; receive a similarity threshold; and generate aset of filters by: forming said set of filters based on a set ofcombinations of positions of matches and gaps for said size of saidsearch term according to said similarity threshold, wherein each one ofsaid combinations contains at least one filter from said set of filters.2. The computer system of claim 1, wherein said at least one hardwareprocessor is further configured to create a minimal set of filters by:determining, for each filter of said set of filters, that said filter isa retainable filter when there exists at least one combination of saidset of combinations which contains only said filter from said set offilters, and deleting said filter from said set of filters when saidfilter is not a retainable filter.
 3. The computer system of claim 1,wherein each combination of said set of combinations includes a match atthe first position of said respective combination.
 4. The computersystem of claim 3, wherein each combination of said set of combinationsincludes a match at the last position of said respective combination. 5.The computer system of claim 1, wherein said at least one hardwareprocessor is configured to form said set of filters by: initializingsaid set of filters; for each selected combination of said set ofcombinations: searching for an occurrence of a filter of said set offilters in said selected combination, deriving a new filter from saidselected combination when said occurrence of a filter of said set offilters is not found in said selected combination, and adding said newfilter to said set of fillers.
 6. The computer system of claim 5,wherein said new filter is a segment of said selected combination. 7.The computer system of claim 5, wherein any of said new filter and anumber of matches of said new filter is determined with a randomizationprocess.
 8. The computer system of claim 7, wherein said number ofmatches is a predetermined quantity.
 9. The computer system of claim 5,wherein said at least one hardware processor is configured to derivemultiple new filters from said selected combination and assign a scoreto each of said multiple new filters, and add said new filter to saidset of filters when said score of said new filter is higher than saidscore of any other new filter of said multiple new filters.
 10. Thecomputer system of claim 9, wherein said at least one hardware processoris further configured to assign said score to each filter of said set offilters in accordance with a parameter selected from the groupconsisting of: respective of applying said filter, an expected number ofcomparisons between said search term and a text; a frequency ofoccurrence of said filter in said set of combinations; a number ofmatches in said filter; and a span of said filter.
 11. The computersystem of claim 10, wherein said at least one hardware processor isfurther configured to: create a second set of filters by: select filtersfrom said set of filters, replace each said selected filters with atleast one different filter, said selecting and replacing performedaccording to a sequence sorted from the highest score of said scores ofsaid selected filters to the lowest score of said scores of saidselected filler, wherein each one of said combinations contains at leastone filter from said second set of filters, and select said second setof filters over said sets of filters in accordance with a fitness score.12. The computer system of claim 11, wherein said fitness score iscalculated using a parameter selected from the group consisting of: anexpected computation speed of search determined as a function of a sumof an expected number of comparisons for all said filters in said set offilters; an amount of memory required to conduct a search respective ofsaid set of filters.
 13. The computer system of claim 1, wherein said atleast one hardware processor is further configured to generate multipleones of said set of filters, wherein: each position in said search termcorresponds to at least one of said multiple ones of said set offilters, each one of said combinations contains, at a position in saidrespective one combination, at least one filter from at least one ofsaid multiple ones of said set of filters, said position of said searchterm corresponding to said at least one of said multiple ones of saidset of filters corresponding to said position in said respective onecombination.
 14. The computer system of claim 13, wherein said at leastone hardware processor is configured to form each of said multiple onesof said set of filters by: designating said set of combinations as acurrent set of combination, and designating said set of filter based onsaid current set of combinations as a current set of filters, obtaininga sequence of positions in said search term, for each said position, inaccordance with said sequence, generate a set of filters in associationwith said position, update a combination from said current set ofcombinations when said combination contains at said position a filterfrom said generated set of filters associated with said position, deletea filter from said current set of filters when said filter is notretainable with respect to said current set of combinations, whereinsaid filter is retainable with respect to said current set ofcombinations when there exists at least one combination of said currentset of combinations which contains only said filter.
 15. The computersystem of claim 14, wherein said at least one hardware processor isconfigured to form said set of filters associated with a position by:obtaining said set of combinations, said current set of filters and saidposition; for each filter of the said current set of filters: add afilter to said set of filters associated with said position when saidfilter is found in at least one of said combinations from said set ofcombinations at said position, and delete said at least one of saidcombinations from said set of combinations.
 16. The computer system ofclaim 1, wherein said at least one hardware processor is furtherconfigured to: receive a lossy threshold, create a subset of saidcombinations in accordance with said lossy threshold respective of saidsearch term and said similarity threshold, wherein generating said setof filters comprises generating said filters respective of said subsetof said combinations.
 17. The computer system of claim 1, wherein saidsize of said search term corresponds to multiple possible sizes for saidsearch term, and wherein said similarity threshold corresponds tomultiple similarity thresholds, each similarity threshold correspondingto a size from said multiple possible sizes for said search term. 18.The computer system of claim 15, wherein said at least one hardwareprocessor is further configured to form a plurality of sets of sets offilters, wherein each one of said plurality of sets of sets of filterscorresponds to a different sequence of a plurality of sequences of saidpositions in said search term.
 19. The computer system of claim 18,wherein said at least one hardware processor is further configured toselect a set of sets of filters from said plurality of sets of sets offilters according to an expected number of comparisons performed whensaid selected set of sets of filters is applied for executing saidk-mismatch search, wherein said expected number of comparisons is afunction of a given text and a given word for search.
 20. The computersystem of claim 1, wherein said at least one hardware processor isfurther configured to apply said executing of said k-mismatch search forclustering a plurality of words of a text.
 21. The computer system ofclaim 20, wherein said at least one hardware processor is furtherconfigured to apply said clustering to accelerate a text indexing stepof said k-mismatch search.